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PREFACE 


This  report  documents  the  results  of  a  preliminary  study  conducted  to  demonstrate  the 
viability  of  the  Training  EfBciency  and  Effectiveness  Methodology  (TEEM).  This  work  was 
performed  under  in-house  Work  Unit  No.  1121-12-00. 

The  authors  are  grateful  to  Major  Marty  Pellum,  Capt  Mary  Knoll  and  Ms.  Michele 
Morales  Olea  for  their  contributions  at  earlier  stages  of  this  project.  A  previous  version  of  this 
paper  was  presented  at  the  annual  meeting  of  the  International  Military  Testing  Association, 
October,  1993. 
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APPLICATION  OF  THE  TRAINING  EFFICIENCY  AND  EFFECTIVENESS 
METHODOLOGY  (TEEM)  TO  AEROSPACE  GROUND  EQUIPMENT 

TECHNICAL  TRAINING 


SUMMARY 

The  Training  Efficiency  and  Effectiveness  Methodology  (TEEM)  was  demonstrated  in  this  initial 
application  to  the  Aerospace  Ground  Equipment  technical  training  course.  Course  effidency  and 
effectiveness  information  were  collected,  summarized  and  displayed  to  demonstrate  how  these 
data  could  be  used  by  course  personnel  to  re-assess  training  needs  and  re-design  the  course. 

L  INTRODUCTION 

A  significant  amount  of  resources  are  devoted  to  initial  technical  training  in  the  military. 
Therefore,  training  that  is  irrelevant  or  inappropriate  for  technical  training  is  inefficient  and  costly 
(Ruck,  Thompson,  &  Stacy,  1987).  Due  to  the  reduction  of  resources  devoted  to  training  and  the 
increased  complexity  of  Air  Force  equipment  and  jobs,  improving  the  quality  of  training  has  never 
been  more  critical.  The  purpose  of  this  paper  is  to  describe  an  initial  research  and  development 
effort  directed  at  assessing  the  efficiency  and  effectiveness  of  Air  Force  training  courses  for  the 
purpose  of  training  course  re-design.  We  call  this  new  approach  the  Training  Efficiency  and 
Effectiveness  Methodology  (TEEM). 

Background 

There  are  multiple  purposes  and  reasons  for  evaluating  training.  The  most  popular 
framework  for  depicting  the  training  evaluation  process  was  developed  by  Kirkpatrick  (1976). 
This  four-level  framework  includes:  student  reactions  to  training;  learning  of  facts,  principles  or 
techniques;  changes  in  on-the-job  behavior;  and  tangible  business  results.  Although  this 
framework  has  advanced  the  practice  of  training  evaluation  over  the  years,  there  is  no  specific 
mechanism  offered  for  using  the  evaluation  information  to  make  changes  to  training  courses. 

However,  training  models,  such  as  those  proposed  by  Goldstein  (1991)  and  the 
Instructional  Systems  Design  (ISD)  process  used  by  the  Air  Force  (AF  Manutd  50-2,  1979), 
depict  evaluation  as  a  key  component  that  provides  feedback  information  to  the  needs  assessment, 
design,  development  and  delivery  stages  of  training  to  continuously  update  and  improve  the  entire 
training  process.  In  most  cases,  training  evaluation  information  is  used  to  determine  if  a  course 
worked  or  not  (e.g.,  Kirkpatrick's  criteria),  and  whether  individual  students  passed  or  failed  the 
course.  These  training  models  suggest  a  more  important  role  for  evaluation  information. 
Evaluation  information  should  be  instrumental  in  helping  training  managers,  course  designers, 
developers  and  instructors  make  informed,  accurate  and  meaningfiil  changes  to  the  course. 

Despite  these  conceptual  models,  little  research  has  been  conducted  to  develop 
methodologies  for  using  training  evaluation  information  to  make  training  course  changes.  One 
exception  is  Ford  &  Wroten  (1984),  who  developed  a  Matching  Technique  that  compares,  or 
matches,  the  importance  of  a  task  to  job  performance  (i.e.,  recommended  "training  emphasis")  to 
the  emphasis  placed  on  tasks  in  a  training  program  in  order  to  depict  the  content  validity  of 
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training.  Ford  &  Sego  (1990)  recommended  that  this  technique  be  used  to  determine  training 
efiSciency  and  developed  a  conceptual  framework  to  integrate  efficiency  and  effectiveness 
information  for  course  re-design.  The  present  paper  describes  the  initial  effort  to  demonstrate  the 
potential  use  of  this  methodology. 


n.  METHOD,  PROCEDURE  AND  RESULTS 


Participants 

Participants  were  182  Aerospace  Ground  Equipment  (AGE)  trainees  in  the  Airman  Basic- 
in-Residence  (ABR)  course  at  Chanute  AFB,  IL.  In  addition,  the  training  manager,  course 
developers  and  instructors  participated  in  various  stages  of  the  data  collection  process.  The 
course  consisted  of  14  blocks  of  instruction  taking  18  weeks. 

Training  Efficiency 

Training  efficiency  examines  whether  the  appropriate  amount  of  time  is  devoted  to  each 
task  taught  in  training.  This  requires  information  that  "recommends"  the  time  required  to  learn  a 
task  and  the  actual  time  devoted  to  training  each  task.  Occupational  Survey  Reports  (OSRs) 
produced  by  the  Air  Forcje's  Occxipational  Measurement  Squadron  (OMS)  contain  a  rich  source  of 
information  that  is  used  to  identify  and  prioritize  tasks  to  be  included  in  initial  technical  training 
courses.  From  the  OSR,  the  task  learning  difficulty  rating  was  used  as  the  measure  of  the 
recommended  time  to  learn  a  task.  Since  this  index  is  based  on  the  notion  of  task  learning  time,  it 
provides  a  reasonable  estimate  of  the  relative  time  that  should  be  devoted  to  training  each  task. 
The  construct  validity  of  the  task  learning  difficulty  ratings  have  been  well  established  against 
entry-level  training  variables  (Mumford,  Weeks,  Harding,  &  Fleishman,  1987)  and  job 
performance  measures  (Dickinson  &  Teachout,  1993). 

A  four-step  procedure  was  developed  to  collect  information  on  the  actual  time  (in  hours) 
devoted  to  each  task  taught  in  training.  First,  course  personnel  determined  which  OSR  tasks  were 
taught  in  the  course.  One-hundred  and  fifty-two  of  the  1016  total  tasks  were  taught  in  the  ABR 
course.  The  second  step  required  instructors  to  link  these  152  tasks  to  specific  training  objectives 
in  the  course  Plan  of  Instruction  (POI).  In  several  cases,  a  single  task  was  covered  in  more  than 
one  training  objective.  The  third  step  required  instructors  to  divide  the  time  allocated  for  each 
trmning  objective  among  all  of  the  tasks  covered  in  the  objective.  For  example,  if  an  objective 
contains  6  hours  of  instruction  and  3  tasks  are  covered  in  that  objective,  the  instructors  estimated 
how  much  of  the  6  hours  was  spent  on  instruction  for  each  task.  The  fourth  step  was  to  sum  the 
amount  of  training  time  devoted  to  each  task.  Since  tasks  could  be  covered  by  more  than  one 
training  objective,  the  time  for  each  task  was  summed  to  create  a  single  number  for  the  amount  of 
time  spent  training  that  task  in  the  entire  course. 
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Before  the  task  learning  difficulty  rating  and  the  hours  spent  in  trmning  could  be  matched 
for  each  task,  both  variables  were  transformed  into  standard  scores  with  a  mean  of  zero  and  a 
standard  deviation  of  one.  Ford  and  Wroten's  (1984)  Matching  Technique  was  then  used  to 
"match"  the  two  data  elements  and  the  results  were  plotted  for  each  task. 

For  demonstration  purposes.  Figure  1  displays  the  results  of  these  matches  for  10  tasks. 
These  tasks  were  chosen  to  represent  different  content  areas  (i.e.,  AGE  equipment)  and  different 
levels  of  task  learning  difBculty.  The  task  learning  difficulty  rating  (i.e.,  relative  time  to  learn)  is 
represented  on  the  x-axis,  the  actual  course  time  spent  training  each  task  is  represented  on  the  y- 
axis,  and  the  resulting  match  is  represented  in  the  figure  by  the  OSR  task  numbers.  Three 
potential  outcomes  are  depicted.  Training  matches  result  when  the  difference  between  the  two 
variables  is  small  (i.e.,  the  standard  score  difference  is  1  standard  deviation  or  less).  Thus,  these 
task  numbers  are  displayed  between  the  diagonal  lines  on  the  graph.  When  the  task  learning 
difficulty  variable  is  greater  than  the  corresponding  actual  training  time  variable  (i.e.,  the  task 
learning  difficulty  standard  score  exceeds  the  actual  task  time  standard  score  by  more  than  1 
standard  deviation),  the  task  number  falls  in  the  bottom-right  of  the  graph,  indicating  a  potentially 
under-trained  task.  Alternatively,  when  relatively  more  time  is  spent  training  a  task,  compared  to 
the  corresponding  task  learning  difficulty  variable  (i.e.,  the  actual  training  time  standard  score 
exceeds  the  corresponding  task  learning  difficulty  standard  score  by  more  than  1  standard 
deviation),  the  OSR  task  number  is  displayed  in  the  upper-left  of  the  graph,  indicating  a 
potentially  over-trained  task. 


ACTUAL  COURSE 
EMPHASIS 

(TIME  IN  HOURS) 


OSR  TASK  LEARNING  DIFFICULTY 

(TIME  TO  LEARN) 

Figure  1.  Matching  Technique  to  Assess  Training  Efliciency. 
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Results  were  found  for  each  of  the  three  potential  outcomes.  For  example,  Task  559,  Test 
Cylinder  Compression,  is  a  match,  indicating  that  the  relative  amount  of  time  spent  in  training 
seems  appropriate.  However,  Task  381,  Isolate  Electrical  Circuitry  Malfunctions,  is  potentially 
over-trained  task,  while  Task  261,  Perform  Gtenerator  Service  Inspections,  is  a  potentially  under¬ 
trained  task. ' 

Training  Effectiveness 

Training  effectiveness  determines  whether  trainees  learned  or  can  perform  the  tasks  taught 
in  training.  This  requires  information  about  the  knowledge  or  performance  levels  of  trainees  at 
the  end  of  the  training  program  and  later  on  the  job.  The  greater  the  learning  and  performance, 
compared  to  a  specified  standard,  the  greater  the  effectiveness  of  the  training  program.  In  this 
study,  training  effectiveness  was  examined  for  knowledge,  only  at  the  end  of  the  training  program. 

An  end  of  course  knowledge  test  was  developed  to  determine  whether  trainees  learned  the 
material  that  was  taught  in  training.  Between  6  and  12  multiple-choice  test  items  were  developed 
for  the  each  of  the  10  tasks  referred  to  above.  This  test  was  administered  to  152  trainees  after 
they  completed  the  ABR  course.  These  knowledge  test  scores  were  used  as  the  effectiveness 
measure.  A  70%  criterion  was  used  for  the  knowledge  items  associated  with  each  task.  For  this 
study,  knowledge  level  was  dichotomized  as  low  (i.e.,  below  standard)  or  high  (i.e.,  above 
standard). 

Figure  2  displays  the  efficiency  and  effectivenss  integration  as  a  3  X  2  matrix.  Overall, 
when  performance  is  above  standard,  it  is  less  likely  that  course  rewsions  are  necessary.  Using 
the  previous  examples.  Task  559  is  a  Training  Match  that  graduates  learned  above  standard. 
However,  Task  381  is  a  potentially  Over-Trained  Task  that  graduates  learned  above  standard. 

This  is  a  potential  area  where  training  time  could  be  reduced  if  the  course  was  shortened  or  if 
addtional  time  is  required  in  another  area  of  the  course.  On  the  other  hand,  course  revisions  are 
more  likely,  if  not  required,  whenever  performance  is  below  standard.  Again,  using  the  previous 
example.  Task  261,  is  a  potentially  Under-Trained  Task,  that  graduates  learned  below  standard. 

A  potential  revision  here  might  be  to  add  training  time  to  this  area.  Of  course,  other  options 
should  always  be  considered,  such  as  developing  improved  instructional  approaches. 
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HIGH 
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488 

177 

788 

381 

559 

331 

286 

431 

261 

229 

Figure  2.  Training  Efficiency  and  Effectiveness  Integration. 


m.  DISCUSSION  AND  SUMMARY 

This  paper  demonstrated  the  potential  use  of  the  TEEM  for  training  program  re-design. 
In  general,  the  results  illustrate  how  training  time  might  be  reduced  for  over-trained  tasks  that 
were  performed  well,  while  training  time  might  be  increased  for  under-trained  tasks  that  were 
performed  poorly.  Thus,  this  information  might  be  be  utilized  to  facilitate  traimng  course 
changes.  In  the  Air  Force,  training  personnel  frequently  make  training  course  revisions  due  to 
reductions  in  course  length,  additional  requirements  dictated  by  policy  changes,  or  customer 
feedback  that  suggests  that  an  area  of  training  is  deficient.  This  methodology  utilizes  relevant 
information,  integrates  that  information  in  a  way  that  is  easy  to  use  and  understand,  and  puts  it  in 
a  format  that  can  facilitate  decision-making  by  trmning  course  personnel. 

Only  10  tasks  were  displayed  here  for  demonstration  purposes.  For  the  course  personnel, 
the  Matching  Technique  was  used  to  display  results  for  the  tasks  associated  with  each  block  of 
instruction.  Thus,  all  152  tasks  in  the  course  were  matched  and  graphed.  Although  some 
potential  benefits  of  integrating  key  evaluation  information  was  demonstrated,  it  is  too  time 
consuming  and  obtrusive  to  develop  and  administer  multiple  knowledge  test  items  that  tap 
individual  task  knowledge.  In  additon,  since  all  trainees  must  meet  the  knowledge  and 
performance  standards  to  complete  training,  it  is  unlikely  that  task-level  effectiveness  information 
in  the  course  is  worthwhile.  All  tasks  would  fall  in  the  "above  standard"  column  of  the 
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would-provide  any  added  value,  since  all  tasks  would  fall  in  the  “above  standard"  column  of  the 
effectiveness  and  efiBciency  integration.  Therefore,  in  future  applications  we  will  focus  on  job 
effectiveness  information  from  supervisory  performance  ratings.  This  provides  feedback  directly 
from  the  field  personnel  that  utilize  the  course  graduates  and  there  is  likely  to  be  some  tasks  that 
fall  below  the  trained  standard. 

We  also  plan  to  collect  data  from  graduates  on  early  on-the-job  opportunities  to  perform 
(OTP)  trained  tasks  (Ford,  Quinnones,  Sego,  &  Sorra,  1992).  By  examining  when  graduates  first 
begin  performing  the  trained  tasks,  along  with  the  amount  and  type  of  practice  they  receive, 
training  transfer  can  be  assessed  and  used  to  facilitate  changes  to  training  courses.  For  example,  if 
there  is  considerable  time  devoted  to  training  a  task,  yet  relatively  few  graduates  report  actually 
performing  the  task  in  their  first  year  on  the  job,  training  personnel  might  consider  reducing  the 
time  spent  training  that  task  and  allocate  that  time  to  other  tasks  in  the  course.  Finally,  efforts 
are  in  progress  to  develop  IBM  compatible  software  that  will  read  in  data,  integrate,  display  and 
print  the  results  such  as  those  presented  in  this  paper. 

In  summary,  this  methodology  is  a  simple,  rational  approach  that  facilitates  training  course 
revisions  by  linking  training  content,  training  efiBciency  and  training  effectiveness  information  in  a 
format  that  is  easy  to  use  and  understand,  and  importantly,  will  facilitate  decision-making  by 
training  course  persormel.  Conceptual  models  of  trmning  (e.g.,  Goldstein,  1991)  suggest  that 
evaluation  plays  a  vital  role  in  helping  training  personnel  re-assess  training  needs  and  re-allocate 
training  resources.  This  methodology  helps  us  take  a  step  in  that  direction. 
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